Multimodality and Gestures in the Teacher?s Communication

The paper presents a research on the multimodal communication of teachers in the classroom. The “musical score”, a procedure for the analysis of multimodal communication, is used to analyse aspects of the signals produced and of the meanings conveyed by teachers while interacting with their pupils, by focusing not only on affective and interactional aspects, but also on the cognitive effects of nonverbal communication. Finally, the paper shows how, based on this procedure, it is possible to analyse chunks of teachers’ communication, and to distinguish different multimodal communicative styles among different teachers. 1 Multimodality in the Classroom Among studies on everyday interaction, an interesting area for the study of multimodal communication is teacher-pupil interaction, a topic generally studied in the domains of education, linguistics, sociolinguistics, pragmatics, and social psychology [1], [2], [3], [4]. [5] applied Ekman and Friesen’s analysis [6], mainly stressing the affective and interactional aspects of the teacher’s nonverbal communication. But the action of multimodality is not limited to affective issues: it has important effects in at least three domains of educational interaction: a. Conversational interaction The teacher’s facial, gestural and bodily communication has an important role in turntaking management and in providing feed-back to pupils. b. Cognitive import Gestures sometimes convey conceptual, concrete and abstract information [7], [8]; and even gaze signals may convey propositional concrete and abstract contents [9]. Moreover, gesture, gaze and posture may help students to understand the teacher’s discourse [10]. c. Teacher’s prejudice As shown by [4], the “Pygmalion effect”, a typical case of a teacher’s prejudice that affects a pupil’s performance, is caused at a large extent by the feed-back the teacher provides to the pupil, which is mainly conveyed by his/her multimodal communication. All of these aspects of teacher-pupil interaction point at the necessity of studying the teacher’s multimodal communication 1. to enhance teachers’ self-consciousness, selfmodeling and regulation, 2. to develop Pedagogical Agents, for simulation purposes [11]. Scholars at the crossroads of Psychology, Artificial Intelligence, Autonomous Agents and Computer Graphics engage in building Embodied Believable Conversational Agents [12], [13], Artificial Agents able to interact with the User through voice, facial expression and gaze, gesture and body posture. In this view, studying the human teacher’s communicative behaviour is a necessary step to discover the rules that govern her or his multimodal communication, not only because it is useful in building Artificial Teachers (would we all like to learn things only from a virtual Mentor?), but also because the approach of simulation, the effort to reconstruct human intelligence and behaviour in machines, is one of the best means to obtain a deeper knowledge of human behaviour. For both these purposes, one has to single out the most typical features of teachers’ multimodal communication: what verbal and nonverbal signals they use most frequently, what meanings they convey, and how they combine signals in the different modalities, to communicate the meanings required in their educational interaction. 2 A Human’s Meanings To find out the specific features of the teacher’s communication, we start from a theoretical model of communication that makes some hypotheses about all the meanings that human need in general to communicate, and the verbal and nonverbal signals devoted to conveying them, and we predict what meanings a teacher might specifically need to communicate most frequently. Finally, we test these predictions by analysing real data collected in empirical research on teachers’ multimodal communication. We distinguish three classes of meanings Humans generally convey during communication: Information on the World, on the Speaker’s Identity, and on the Speaker’s Mind [14]. Information on the World. As we talk we provide information about the concrete and abstract events we mention, their actors and objects, and the time and space relations among them. This is provided mainly through the words of sentences and their syntactic structure; but often also by deictic, iconic and symbolic gestures. A deictic gesture indicates something in the surrounding environment: a way to set the reference of our discourse, then a way to explain what, in the external world, we are going to talk about. An iconic gesture instead describes the shape, size or movements of some referent we are mentioning; and this description can be sometimes metaphorically extended to refer to some abstract referent. Finally, some symbolic gestures directly mention some object, feature or action. But not only gesture can indicate or describe; sometimes we point at things or persons by eye, lip or chin direction, and we may refer to features of objects or persons also by gaze, prosody and body movement: we squeeze our eyes to refer to something small or difficult, we open eyes wide to refer to something huge, we utter a longer vowel to say something is long, we speak in a “staccato” way to indicate precision, we refer to a person we know by moving as she does. And also a teacher can add these kinds of information to her verbal discourse by using hands, voice and gaze. Information on the Speaker’s Identity. As we talk, physiognomic traits of our face, eyes, lips, the acoustic features of our voice, and often our posture, provide information on our sex, age, socio-cultural roots, and personality. But besides providing information on our objective identity, we also inform about the Identity we claim is ours: our image, that is, how we want others to see us. In other words, in any moment of our life and communication we have a goal of Self-presentation: we inform not only about how we are, but about how we want to be seen. This is often done without awareness; but a teacher, at her first class with new pupils, might monitor her appearance and behaviour to project a particular image of herself, to set the basis for future interaction. Information on the Speaker's Mind. While talking about events of the external world, we also communicate why we want to talk about those events, what we think and feel about them, how we plan to talk about them: we provide information on the beliefs we’re mentioning, our goals concerning how to talk about them, and the emotions we feel while talking [15]. a. Among information concerning our own beliefs, we inform: 1. on the degree of certainty of the beliefs we are mentioning, by words like perhaps, certainly, or the conditional or subjunctive verb mode, but also by frowning, which means: “I am serious in stating this” [10], or by open hands palms up, which means “this is self-evident”[16]; 2. on the source of the beliefs we mention, whether they come from memory, inference, or communication [17]: we look up when trying to make inferences, snap fingers while trying to remember, we make the gesture “quote” with index and middle fingers curved twice to mean that we are quoting other people’s words for which we are not responsible. b. While talking we inform about our goals concerning our sentence (b.1., b.2.), our discourse (b.3.), and the ongoing conversation (b.4.), then about: 1. the performative of our sentence, that may be conveyed by performative verbs [18], [19] but also through intonation or performative facial expression [20]; 2. the topic-comment distinction within a sentence or discourse, which may be conveyed by batons, eyebrow raising, high intensity or pitch of a tonic vowel; 3. the discourse rhetorical relationships: a list may be scanned by words (first, second, third...), but also by counting on fingers, or marking all the items in the list with the same intonational contour; topic shift may be expressed through posture shift; 4. the turn-taking and back-channel structure of conversation: we raise our hand to ask for turn; we nod to reassure the Interlocutor we are following, understanding, approving of what he’s saying. c. Finally, we inform on the emotions we are feeling while talking, not only by affective words, but with gestures, emotional intonation, facial expression, gaze and posture. 3 A Teacher’s Meanings A teacher in classroom interaction may convey all of these meanings; but some are conveyed more frequently, because of the teacher’s particular role. Starting from the taxonomy above, then, we can predict the specific communicative profile of a teacher’s communication. Of course these predictions stem from our stereotype of the teacher’s role, but they can be tested on real data. 3.1 Information on the world. A great part of the teacher’s classroom communication informs about the contents of different disciplines, that include both concrete and abstract information. In some cases, concrete information like the shape of an object or a space, or a sequence of actions is better conveyed by iconic signals, like pictures, paintings, pantomime or iconic gestures. But gestures have proved to be useful also in the explanation of abstract (for example, mathematical) concepts [8]. 3.2 Information on the Speaker’s Mind. This is the realm in which teacher’s multimodal communication is most informative. Beliefs. Starting from information on the Speaker’s beliefs, namely the degree of certainty of the beliefs she mentions (a.1.), we predict a teacher will not convey uncertainty or doubt so often, given the image of a self-confident person she must generally project in front of her pupils. So, her most frequent expression will be the small frown showing that she is serious in what she says, and that she believes it strongly. As to the metacognitive information about the source of her current knowledge (a.2.), during a lecture the teacher will rarely gaze side-downward, or raise eyes up, because she is usually supposed to remember the things she reports very well, without needing to retrieve them from long-term memory, or to reason long about them. Goals. The teacher’s communication (as opposed to the layperson’s) is supposedly particularly rich in providing information about her sentence, discourse and conversational goals. As shown by studies on teachers’ behaviour [1], the teacher’s performatives, i.e. the specific communicative intentions of her sentences, are a great number: question, peremptory order, advice, approval, praise, blame, reproach, question with an indirect meaning of command, praise with an ironic meaning of blame; but they are rarely conveyed by the classical performative verbs (I order, I advice, I propose... [18], [19]); they are generally expressed by intonation, gaze or facial expression. Instead of saying “I order you to turn off you cellular”, the teacher will express the propositional content of the speech act (“Turn off your cellular”) while conveying its performative by a severe directive intonation, and/or by a serious face, no smile, head up, gaze staring at he pupil. Performative intonation and performative facial expression [20] are sophisticated devices of human communication, devoted to convey even the most subtle nuances that distinguish performatives from one another. This is why the teacher’s voice and face behaviour are important in educational interaction. One more frequent and important information the teacher conveys, but mostly not by verbal language, is about the relative importance of the different parts of a sentence of discourse: the topic-comment distinction. More than in everyday communication, the teacher needs to distinguish, within her sentence or discourse, the beliefs that pupils have to retrieve from their long-term memory (topic) from the new beliefs to connect with them that are the object of the teacher’s present communication (comment). Often, to explain new concepts that to be understood require reference to previous contents, the teacher has to recall a lecture or an activity performed even months before. This distinction between what is to be retrieved from memory and what is coming as new is generally expressed by facial and intonational devices [10]. Again, gesture and intonation are of help when the teacher makes explicit the rhetorical relationships among different parts of the discourse she is delivering. An aid to understand and memorise a discourse is to know the Author’s outline. This is why a teacher often says: “I’ll now speak of X... now I come to topic Y...”. But this can also be done in nonverbal ways: a posture shift, for instance, signals we are moving to another topic or starting a digression [21]. Another way to stress transition to another sub-topic is counting paragraphs, not only verbally (“first... second.... third...”) but also gesturally, by counting on fingers. Intonation has an important role too: a recurrent intonational contour signals we are listing different items of the same class, while we can mark a parenthetic clause through lower pitch and intensity [22]. The signals to regulate turn-taking and to provide back-channel are particularly important in classroom interaction, both for teacher and pupils; and the teacher must be particularly aware of these signals, whether delivered from herself or the pupils. A trivial example is when a pupil raises a hand to ask for a speaking turn. But while this holds in a classroom with a quite asymmetrical structure and rigid interactional norms, it could not hold in a democratic group or while working in a small team. In this case, a teacher that wants all members to talk must be aware of more subtle ways of asking for a speaking turn: leaning forward, opening mouth as in starting to speak, opening eyes wider. Finally, a back-channel signal is any communicative verbal or nonverbal act performed by an Interlocutor to provide a feed-back to the present Speaker about whether the Interlocutor is a. understanding, b. believing, c. approving, and d. finding interesting what the Speaker is saying. I can say “I see” or simply nod to tell you I understand what you are saying; if I don’t understand I’ll say “I can’t follow you” or frown; if I understand but I don’t believe it, I can say “I don’t believe it”, shake my head or show a facial expression of doubt or perplexity; I can nod also for approving, and shake head, shake index finger, or frown for disapproving; if what you say is very interesting I’ll say “You don’t say!”, or “Oh!” emphatically, or raise eyebrows to show surprise; but if I am not very much concerned or definitely bored, I can cyclically repeat “oh”, with a very flat intonation, or even yawn. Now, some back-channel signals may be emotionally loaded, since they imply an evaluation and then may touch or hurt the Speaker’s image or self-image: letting the speaker know that I don’t believe what she says, or that her talk is boring, may be offensive. So we can predict 1. what back-channel signals are more frequently provided by teachers and pupils, respectively, and 2. what back-channel signals are more relevant for, respectively, teachers’ and pupils’ behaviour. A very young pupil will not perform many back-channel signals of incredulity, not only because it is implausible for him to have a knowledge base sufficiently larger than the teacher’s enabling him to know things contrasting with what she says; but also because, even if the pupil really does not believe the teacher, he might be afraid to offend her by showing it. Again, a pupil will be particularly sensitive to back-channel signals of approval and disapproval. Conversely, if a teacher thinks motivation is an important basis for learning, she will pay attention to signals of interest or boredom. Emotions. As to information about emotions, a teacher may feel various kinds of emotions during class work: joy, stress, anxiety, enthusiasm; tenderness, compassion, love, hatred, worry, anger, interest, curiosity, boredom. Of course, the very fact that a teacher expresses her emotions may be subject to sanction; so some teachers may claim they do not express their emotions to pupils, but yet their emotions can leak out from nonverbal behavior: anxiety or anger may be not expressed by a specific signal, but by some aspects of how one performs verbal or nonverbal signals: some subparameters of gestural or bodily behaviour like rhythm or muscular tension may vary producing a particular muscular tension in making gestures, a higher pitch of voice in speaking, a higher speed in moving around in the classroom. 3.3 Information on the Speaker’s Identity The teacher’s multimodal communication provides information about her identity: the image she wants to project of herself as an easy, serious, rigorous teacher may be conveyed by her exterior look (clothing, hairdo) but also, once more, by the way she produces verbal and nonverbal signals: smooth movements may provide an image of a relaxed and easy-going person, jerky movements tell of a hectic behaviour. 4 The musical score of the teacher’s communication Testing our predictions about the teacher’s multimodal communication requires us to assess the quantity and quality of multimodal communication in different teachers and to single out the communicative and teaching style of different teachers. To this goal, a tool is needed that is capable of showing how many and what meanings the teacher conveys, but also how these different meanings are related to each other. In a previous work, [14] proposed the “musical score”, a procedure for the transcription, analysis and classification of signals and their interaction in multimodal communication. This procedure (Poggi, Pelachaud & Magno, this volume), has been applied to the study of the teacher’s multimodal communication. 5 A research study on the teacher’s multimodal communication. We conducted a research study on the teacher’s gestural communication in four classes of a Primary School in a village of the province of Rome, Italy.